{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "\n",
    "<h1 align=\"middle\"> Logistic Regression on notMNIST Images </h1> <br>\n",
    "Download Data Set Directory: http://yaroslavvb.com/upload/notMNIST/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 align=\"left\"> Download Needed Data Files </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# These are all the modules we'll be using later. Make sure you can import them\n",
    "# before proceeding further.\n",
    "from __future__ import print_function\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import os\n",
    "import sys\n",
    "import tarfile\n",
    "from IPython.display import display, Image\n",
    "from scipy import ndimage\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from six.moves.urllib.request import urlretrieve\n",
    "from six.moves import cPickle as pickle\n",
    "import random\n",
    "import hashlib\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2016-09-03 10:13:27--  http://yaroslavvb.com/upload/notMNIST/notMNIST_small.tar.gz\n",
      "Resolving yaroslavvb.com (yaroslavvb.com)... 129.121.4.193\n",
      "Connecting to yaroslavvb.com (yaroslavvb.com)|129.121.4.193|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 8458043 (8.1M) [application/x-gzip]\n",
      "Saving to: ‘data/notMNIST_small.tar.gz’\n",
      "\n",
      "notMNIST_small.tar. 100%[=====================>]   8.07M  1.68MB/s   in 4.9s   \n",
      "\n",
      "2016-09-03 10:13:32 (1.66 MB/s) - ‘data/notMNIST_small.tar.gz’ saved [8458043/8458043]\n",
      "\n",
      "--2016-09-03 10:13:32--  http://yaroslavvb.com/upload/notMNIST/notMNIST_large.tar.gz\n",
      "Reusing existing connection to yaroslavvb.com:80.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 247336696 (236M) [application/x-gzip]\n",
      "Saving to: ‘data/notMNIST_large.tar.gz’\n",
      "\n",
      "notMNIST_large.tar. 100%[=====================>] 235.88M   770KB/s   in 2m 34s \n",
      "\n",
      "2016-09-03 10:16:06 (1.53 MB/s) - ‘data/notMNIST_large.tar.gz’ saved [247336696/247336696]\n",
      "\n",
      "FINISHED --2016-09-03 10:16:06--\n",
      "Total wall clock time: 2m 39s\n",
      "Downloaded: 2 files, 244M in 2m 39s (1.53 MB/s)\n"
     ]
    }
   ],
   "source": [
    "# http://www.thegeekstuff.com/2009/09/the-ultimate-wget-download-guide-with-15-awesome-examples\n",
    "!mkdir -p data && wget -P data http://yaroslavvb.com/upload/notMNIST/notMNIST_small.tar.gz \\\n",
    "      http://yaroslavvb.com/upload/notMNIST/notMNIST_large.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we'll download the dataset to our local machine. The data consists of characters rendered in a variety of fonts on a 28x28 image. The labels are limited to 'A' through 'J' (10 classes). The training set has about 500k and the testset 19.000 labelled examples. Given these sizes, it should be possible to train models quickly on any machine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract the dataset from the compressed .tar.gz file. This should give you a set of directories, labelled A through J."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "notMNIST_large already present - Skipping extraction of notMNIST_large.tar.gz.\n",
      "['notMNIST_large/A', 'notMNIST_large/B', 'notMNIST_large/C', 'notMNIST_large/D', 'notMNIST_large/E', 'notMNIST_large/F', 'notMNIST_large/G', 'notMNIST_large/H', 'notMNIST_large/I', 'notMNIST_large/J']\n",
      "notMNIST_small already present - Skipping extraction of notMNIST_small.tar.gz.\n",
      "['notMNIST_small/A', 'notMNIST_small/B', 'notMNIST_small/C', 'notMNIST_small/D', 'notMNIST_small/E', 'notMNIST_small/F', 'notMNIST_small/G', 'notMNIST_small/H', 'notMNIST_small/I', 'notMNIST_small/J']\n"
     ]
    }
   ],
   "source": [
    "num_classes = 10\n",
    "np.random.seed(133)\n",
    "\n",
    "def maybe_extract(filename, force=False):\n",
    "    root = os.path.splitext(os.path.splitext(filename)[0])[0]  # remove .tar.gz\n",
    "    if os.path.isdir(root) and not force:\n",
    "    # You may override by setting force=True.\n",
    "        print('%s already present - Skipping extraction of %s.' % (root, filename))\n",
    "    else:\n",
    "        print('Extracting data for %s. This may take a while. Please wait.' % root)\n",
    "        tar = tarfile.open(filename)\n",
    "        sys.stdout.flush()\n",
    "        tar.extractall()\n",
    "        tar.close()\n",
    "    data_folders = [os.path.join(root, d) for d in sorted(os.listdir(root))\n",
    "        if os.path.isdir(os.path.join(root, d))]\n",
    "    if len(data_folders) != num_classes:\n",
    "        raise Exception(\n",
    "            'Expected %d folders, one per class. Found %d instead.' % (\n",
    "            num_classes, len(data_folders)))\n",
    "    print(data_folders)\n",
    "    return data_folders\n",
    "  \n",
    "train_folders = maybe_extract('notMNIST_large.tar.gz')\n",
    "test_folders = maybe_extract('notMNIST_small.tar.gz')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's load the data in a more manageable format. Since, depending on your computer setup you might not be able to fit it all in memory, we'll load each class into a separate dataset, store them on disk and curate them independently. Later we'll merge them into a single dataset of manageable size.\n",
    "We'll convert the entire dataset into a 3D array (image index, x, y) of floating point values, normalized to have approximately zero mean and standard deviation ~0.5 to make training easier down the road.\n",
    "A few images might not be readable, we'll just skip them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "notMNIST_large/A.pickle already present - Skipping pickling.\n",
      "notMNIST_large/B.pickle already present - Skipping pickling.\n",
      "notMNIST_large/C.pickle already present - Skipping pickling.\n",
      "notMNIST_large/D.pickle already present - Skipping pickling.\n",
      "notMNIST_large/E.pickle already present - Skipping pickling.\n",
      "notMNIST_large/F.pickle already present - Skipping pickling.\n",
      "notMNIST_large/G.pickle already present - Skipping pickling.\n",
      "notMNIST_large/H.pickle already present - Skipping pickling.\n",
      "notMNIST_large/I.pickle already present - Skipping pickling.\n",
      "notMNIST_large/J.pickle already present - Skipping pickling.\n",
      "notMNIST_small/A.pickle already present - Skipping pickling.\n",
      "notMNIST_small/B.pickle already present - Skipping pickling.\n",
      "notMNIST_small/C.pickle already present - Skipping pickling.\n",
      "notMNIST_small/D.pickle already present - Skipping pickling.\n",
      "notMNIST_small/E.pickle already present - Skipping pickling.\n",
      "notMNIST_small/F.pickle already present - Skipping pickling.\n",
      "notMNIST_small/G.pickle already present - Skipping pickling.\n",
      "notMNIST_small/H.pickle already present - Skipping pickling.\n",
      "notMNIST_small/I.pickle already present - Skipping pickling.\n",
      "notMNIST_small/J.pickle already present - Skipping pickling.\n"
     ]
    }
   ],
   "source": [
    "image_size = 28  # Pixel width and height.\n",
    "pixel_depth = 255.0  # Number of levels per pixel.\n",
    "\n",
    "def load_letter(folder, min_num_images):\n",
    "    \"\"\"Load the data for a single letter label.\"\"\"\n",
    "    image_files = os.listdir(folder)\n",
    "    dataset = np.ndarray(shape=(len(image_files), image_size, image_size),\n",
    "                         dtype=np.float32)\n",
    "    print(folder)\n",
    "    num_images = 0\n",
    "    for image in image_files:\n",
    "        image_file = os.path.join(folder, image)\n",
    "        try:\n",
    "            image_data = (ndimage.imread(image_file).astype(float) - \n",
    "                    pixel_depth / 2) / pixel_depth\n",
    "            if image_data.shape != (image_size, image_size):\n",
    "                raise Exception('Unexpected image shape: %s' % str(image_data.shape))\n",
    "            dataset[num_images, :, :] = image_data\n",
    "            num_images = num_images + 1\n",
    "        except IOError as e:\n",
    "            print('Could not read:', image_file, ':', e, '- it\\'s ok, skipping.')\n",
    "    \n",
    "    dataset = dataset[0:num_images, :, :]\n",
    "    if num_images < min_num_images:\n",
    "        raise Exception('Many fewer images than expected: %d < %d' %\n",
    "                    (num_images, min_num_images))\n",
    "    \n",
    "    print('Full dataset tensor:', dataset.shape)\n",
    "    print('Mean:', np.mean(dataset))\n",
    "    print('Standard deviation:', np.std(dataset))\n",
    "    return dataset\n",
    "        \n",
    "def maybe_pickle(data_folders, min_num_images_per_class, force=False):\n",
    "    dataset_names = []\n",
    "    for folder in data_folders:\n",
    "        set_filename = folder + '.pickle'\n",
    "        dataset_names.append(set_filename)\n",
    "        if os.path.exists(set_filename) and not force:\n",
    "            # You may override by setting force=True.\n",
    "            print('%s already present - Skipping pickling.' % set_filename)\n",
    "        else:\n",
    "            print('Pickling %s.' % set_filename)\n",
    "            dataset = load_letter(folder, min_num_images_per_class)\n",
    "            try:\n",
    "                with open(set_filename, 'wb') as f:\n",
    "                    pickle.dump(dataset, f, pickle.HIGHEST_PROTOCOL)\n",
    "            except Exception as e:\n",
    "                print('Unable to save data to', set_filename, ':', e)\n",
    "    return dataset_names\n",
    "\n",
    "train_datasets = maybe_pickle(train_folders, 45000)\n",
    "test_datasets = maybe_pickle(test_folders, 1800)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 align=\"left\"> Download Needed Data Files </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['notMNIST_large/A', 'notMNIST_large/B', 'notMNIST_large/C', 'notMNIST_large/D', 'notMNIST_large/E', 'notMNIST_large/F', 'notMNIST_large/G', 'notMNIST_large/H', 'notMNIST_large/I', 'notMNIST_large/J']\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAABmUlEQVR4nG2SMW+OURTHf/c+t9VF\nKAnCRGhikQibiYGIkurbrZ9Awmo28ElUIphE4gsYBIvEZGCQlHagYqjX89zzMzzv2/K2Z7v3f87v\n/M/JgT4aBlbVzhUy/0di7xerWv06S+o/x0k2v54SQI4jF3eUZs7XUG1doZkQSfn1iLu2f8TdBuR4\nhECOQ5fSZGnm2E9DbWMXbuZJtD13yy9AKqWUmamFftTqImWyFD5Z//WbgByn7pgwfzhxtzZgWj/9\nPdnnF+6p6sPj1Z47oBmNUtPlbth1w/ry8xsqoDe3Xc4NDcPNuXTLTg3Xx34Lt+20+q5w+IfRc1MD\nGYJ5AHnVTa+9sALBoo42s2Fo9QbT6YpVDdcOkICGZTsNN46S08xHq9o5SA0ZuK4QvF9NkX8/RkAW\nBRL7vlm19X6ZKdPlTI3Y8luYt2pEnOsne26nVpdSKXLNyCTiwWaS7Elyv4dnAXv6bU9EuD4LXIiR\n1rVt27bt+OUgZa6m6Js1pZRSyuhwZEHK292oGq4e5Oyf3SQ1XM7zU7HzIgDCpb+87F/rYwYuuAAA\nAABJRU5ErkJggg==\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAABB0lEQVR4nH3Ru0oEQRAF0FvVM8YG\n4isyMDISs80FExMFA1PBWKP1C1T8AX9ARDBR0HizxUBFhRUDXyz4YEVwAgXd7e5rKDjVU+mhuql7\nwb8JrGGJve/itrE+CVGghIEk+bM/DlUYwxiyhZOpaKKo097AzqCJAJD7idUkwmEujYKxNBJvafQ4\nyBIUQ97aNJCMUMmv5jvGs6JZpo9rtQctb7Jzc986v/gUjWWML+3Lxl1XNNjBR75ujcDB+DOGyKH6\n2WxwBqpzQj96tBgSd6oGt32dTMj5/pWqVqarWhmuaqWowtMq3E0hu+5wz6oMpKCvuWzFJ6KqxcbM\n+7/KCODLfzy1m8fPKvEXrCqNz5Ow7xUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAA1UlEQVR4nLWSv0oCUBSHf+fcAylh\nLjZIg05KL9Aq+Qo+RgTtQeBzNDY0BI3O0ivUFoKEm62B4D3n19Ki3tvmN56P85eDBHTuV8y+S+bi\n5kx9OLarCybdA/3rU3u47bxO+m0X7ED7XgoY3DRRQsVc4YF06BiwBCTAy7nF6PFleRoAmnNpjz8Y\nfK45e5SnN2Gl7GA4Oq84gMFPq0nP3JzUeioQtUTFP4c41m0DqK5pGgKpvAneyXWjXFTE7i6nX4NW\n+TUh6M14QNBfuipJth9reOyB5fznFyzieJh1vzYSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAABcUlEQVR4nHWTMWsUURRGz71vJoXK\nZlItIkLSrKAY9geILjZibEWwtExhoQhqwMIy+AsUkbSSQrCVQKKNlRDQaFSshKSJccwq6yTzPotV\nspE3tz2cW3zfvVa2AECAsbPxZmVlPZrXANbVsSft6P1LZUY+cfLc+cOD1wtPByEKIPBQlcoWBhjF\n3A/p/QWCAT7GnCqVBcFDyIIx/Um17uMAGbf0+58JntPZUKUHOJBxYxRCzlXt7WqWkIA4L7Wr7cnh\n5v/GWMD2iptJKJZ3gnPlaApG//IWqX0xBTE+QqSXhPABGacazG0wjjeYfTDGG+Aw9QZ4BOBnEooC\nxGaDeQKLrDWYHTBepaDHydOYf3+eDv7seB159jkd/DWU9edx2C9yOHm83KvhznqAjNujZVvO1FdV\nmsfBx7i3f2AhOJ13quPdYdOBR6Oneej6lrTaIxhkXdozBLxbZgqtzpmZYrD8eLEKUWAH3kG/vq0u\nvVjT33f4A7d5mVhZacdLAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAApUlEQVR4nJ1SOw5CMQxzSAcYQJyE\nizCxsnOIdz5GBiTEQRBiAhoz8ZoUtRLPUywrPydgA8bzYoYOumIKzEpIZSW6OsQ8iJT9Q0cmtyfc\ngJmrXk9FyQStHsiPVGUS9FQCu7zlG+bl4RgzNyXMum6bYGo/PUP7ehWHVDm0u4+c6YSpDhmjmJE9\n/ePYoSysW9aLgq07NuQajfegDE1vQX01HwyETH/qD5IYZCzrFG8ZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAABYklEQVR4nHWSvUpcURSFv33ONRrx\nBwXBVxAi2MwrCKZP2kC6PEKwMYUPkCptqjQWIUEbixRpFERiE0wnWAQnBBWdOH/37GVxZ8y9d8av\nOM1irXX2ZhsQJHu2vra6NDfDA6n18x0QmXj1rS1J8iGSXKcQaJzIlbq9XBWSW0wvPz5NGAFSzweh\n0wbI2PiapSjj+MuP3/96APjk4YIMWDxVrqTmC/v/GSaa6uV5zhv15bpsEGMMAzLeKyXt8FlJuTaZ\nLDmNqdefdt/Ocy53tVcJlLHiuZVcl3NYVcxCiJbNAHgHVUTlQC1tSMgMIBsruhelAkhnldT+xd6H\ntgk0jlzbxMc6YRY9dPYrSv/P/hYqOmVXa91yhN/eDSaSXH+f1FJjeRSrbShRElXbUEEYOoM9Kgof\nM1UmA8Jyp3U3Jrg4yeuri6Pno96Oe3GI+jVf+zLhwFT4uWmPxK7sNruS1PreGIm9BzW629duU9mw\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAABFklEQVR4nKWSvy5EURDGf3PuXYVs\nIqLyp9iIaDf7AioRJV7DY3iAjUrJE+j3CYRSdESn0FhBQe458yl2z7XcG5H4yvnNzDkz8xEIu2fX\n40o/FTUsfeNkWwZyvhQMgHJz1HOZQUFD5VGv6oDs6e49x6waLMoAniUp6WIVqwsYKSpqWC4AeDh+\nKJVhES1nTTSGVLclkUiFZwh1IRAoCijLFiQu516Dd2+QJFXaaR2lGQLMANQOpUZl8K3T2SV+g0Z3\n/Q9vtrQVb/cOWFqbz0mNUSa7lRRa2+Yf/hvab7AV51GWZkxUHzu8TOoPl6OnrHxAO9+fGuz2o36g\nNlj/UTEmd2+4WoLBlVySxxlNMy14Z++gv9JtMQKfBwi1cVTN/eUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAACQ0lEQVR4nE2STUhUYRSGn/N9914d\n/ImkQaNMcBCZzECxYiZtKIxMpFKygYIkxUIQiyiSFmaLMmqRpYtatbGFixZhq1r4R0RkSERUFkiW\nEhU6Fv6Nzr0t7szUt32/9znveziQfIrWSwhowpUY5LTt+V8Ltc75UYodn/1KNt/x70pqgi/EbC2m\nbHhWgmE+3ErRP1FXm+afMCZ9FVhcb0V2GgmoXfh11YgqVu9FRlOXyypCyvEmjVJlCVN1nPvpE4OR\no+DfF3eKk/87akUH2goYmtRrtesfiRHoT0Y96EXhadp44xom704LTcVxnxJPJQJgfCuB6kgajTUo\nAA2kpgEYxvEvSnh1i4u1riYYuZkuQjPch94WDV/eiwJE0TLlrHShQfDGwtCzcCEX7X7ucl52Xv1+\nBY0hzU6G0bjYjcuEGucUkPE4D1GMjpV09joVKt7QmDiPNizK6zBV1sLHdm/7SrqbHOqHMAVkXTUW\nJ51G+PA8AaXviGgAKyAEZ35Y4rU7SFDVoBMDsJedYLkajlIkIzjxvU3Pu3xrNj8wmP0AAvYb7ITo\ntpWMaOj+iV9PNaWTc5JwzoMCcTYt9a80965pCsfQ8bBqCdsG26cji8fkrqySM05ypkPwULpZun1c\n0XFzTjsp1ntiIe2mXbJeH2hYnB5w7HrdpWxS1makODumbICy/PiZpE/sRglpnwppyYqvyLNf0Fqj\n+pvQIOpFZunh5Iq2VGUCBd0NaMCg5/aZFJW4EDsvCNmeJ291DFC272xHJF70L+IxqvHpX2K+AAAA\nAElFTkSuQmCC\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAA00lEQVR4nO2TPWpCURCFz/w8G9HS\n9KIgWFho6QoshJQp3EFWmSJVBEGwEWwUREnAxkZ9c1I8jfeBC0jhaW7xzZ3TfCOt5RmiIkhDBpG9\nusAUIFNKqCKgUh3VXpr9bqX08zT/Wu0OnwAAsf6MSaYDvS4yd1MMDoyCBH/aUHOzW1OGzR2ucS3R\n4gkw7cxL8HGe8B9DQyKY4E8Cd9dUE/I4hoq5FxNiw8WdBfeTGgSA1N+yRrPX8VIZt7Pl5vsDbeYk\nI1I185xk8N2Ji0K0dA4KRrBy+QXB0H4U+5v+4gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAABqUlEQVR4nGVSwU5TURScc8pTFAjB\nwCsb4wKj/oHRxARISuISomu/QRPYaPQXIKyNblzJlp1NrCaugYWJhoaWB1FqxdTaxva+e8ZFKc/e\nzupOJmdO5p4B4rVStd2qFB/HEEAw86RYabUPS6sxsFSlJ0nP6iJEsHjOKwUc03kzM+9YjoConPEj\n3UEO5r0hhz0HuL2M72Jys0kjaY2NCYhgfKNhJI3NzUkRxvNzeat9/VAXAhDOzN+M9WS/WBOo4gy9\nl0Z9PqICqAgAGgEAwonbN2ZR2//0WzAIwcVnSS/Kt+ejCPGa3qfOpd74JtSihJ3Um/n0r9WGbFfq\nNJI0/ng4tJRXHy1dm5LT5N2raihixBPReK7TMmggCogLlyL3x0M1HZpceHB3eqr762B7K5Fg7vJL\n9uB5uhzkwFM6572ZT7v8OaeDchM5GEhS0OiGSV50erbGz3fw31JRgdmt+/dmr3TrX7ZLzUxTDU00\nmxROL1yPcVL++F0I1f4JAQjG1htndVkfQ5h+i9Y/1dvwY6JKVsmDaFATFI76ZU4Koa0gv1ZK2q3D\n96v5c+0fpUgXP71SwDkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "def ipython_display_samples(folders):\n",
    "    print (folders)\n",
    "    for i in folders:\n",
    "        sample = np.random.choice(os.listdir(i), 1)[0]\n",
    "        display(Image(os.path.join(i, sample)))\n",
    "\n",
    "# train folders\n",
    "ipython_display_samples(train_folders)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 align=\"left\"> New </h3>\n",
    "Merge and prune the training data as needed. Depending on your computer setup, you might not be able to fit it all in memory, and you can tune train_size as needed. The labels will be stored into a separate array of integers 0 through 9.\n",
    "Also create a validation dataset for hyperparameter tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training: (200000, 28, 28) (200000,)\n",
      "Validation: (10000, 28, 28) (10000,)\n",
      "Testing: (10000, 28, 28) (10000,)\n"
     ]
    }
   ],
   "source": [
    "def make_arrays(nb_rows, img_size):\n",
    "    if nb_rows:\n",
    "        dataset = np.ndarray((nb_rows, img_size, img_size), dtype=np.float32)\n",
    "        labels = np.ndarray(nb_rows, dtype=np.int32)\n",
    "    else:\n",
    "        dataset, labels = None, None\n",
    "    return dataset, labels\n",
    "\n",
    "def merge_datasets(pickle_files, train_size, valid_size=0):\n",
    "    num_classes = len(pickle_files)\n",
    "    valid_dataset, valid_labels = make_arrays(valid_size, image_size)\n",
    "    train_dataset, train_labels = make_arrays(train_size, image_size)\n",
    "    vsize_per_class = valid_size // num_classes\n",
    "    tsize_per_class = train_size // num_classes\n",
    "    \n",
    "    start_v, start_t = 0, 0\n",
    "    end_v, end_t = vsize_per_class, tsize_per_class\n",
    "    end_l = vsize_per_class+tsize_per_class\n",
    "    for label, pickle_file in enumerate(pickle_files):       \n",
    "        try:\n",
    "            with open(pickle_file, 'rb') as f:\n",
    "                letter_set = pickle.load(f)\n",
    "                # let's shuffle the letters to have random validation and training set\n",
    "                np.random.shuffle(letter_set)\n",
    "            if valid_dataset is not None:\n",
    "                valid_letter = letter_set[:vsize_per_class, :, :]\n",
    "                valid_dataset[start_v:end_v, :, :] = valid_letter\n",
    "                valid_labels[start_v:end_v] = label\n",
    "                start_v += vsize_per_class\n",
    "                end_v += vsize_per_class\n",
    "                    \n",
    "            train_letter = letter_set[vsize_per_class:end_l, :, :]\n",
    "            train_dataset[start_t:end_t, :, :] = train_letter\n",
    "            train_labels[start_t:end_t] = label\n",
    "            start_t += tsize_per_class\n",
    "            end_t += tsize_per_class\n",
    "        except Exception as e:\n",
    "            print('Unable to process data from', pickle_file, ':', e)\n",
    "            raise\n",
    "    \n",
    "    return valid_dataset, valid_labels, train_dataset, train_labels\n",
    "            \n",
    "            \n",
    "train_size = 200000\n",
    "valid_size = 10000\n",
    "test_size = 10000\n",
    "\n",
    "valid_dataset, valid_labels, train_dataset, train_labels = merge_datasets(\n",
    "  train_datasets, train_size, valid_size)\n",
    "_, _, test_dataset, test_labels = merge_datasets(test_datasets, test_size)\n",
    "\n",
    "print('Training:', train_dataset.shape, train_labels.shape)\n",
    "print('Validation:', valid_dataset.shape, valid_labels.shape)\n",
    "print('Testing:', test_dataset.shape, test_labels.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll randomize the data. It's important to have the labels well shuffled for the training and test distributions to match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def randomize(dataset, labels):\n",
    "    permutation = np.random.permutation(labels.shape[0])\n",
    "    shuffled_dataset = dataset[permutation,:,:]\n",
    "    shuffled_labels = labels[permutation]\n",
    "    return shuffled_dataset, shuffled_labels\n",
    "train_dataset, train_labels = randomize(train_dataset, train_labels)\n",
    "test_dataset, test_labels = randomize(test_dataset, test_labels)\n",
    "valid_dataset, valid_labels = randomize(valid_dataset, valid_labels)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 align=\"left\"> Linear Regression </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "regr = LogisticRegression()\n",
    "X_test = test_dataset.reshape(test_dataset.shape[0], 28 * 28)\n",
    "y_test = test_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 align='Left'>Note on the Following Training Sample Code</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code uses score which returns the mean accuracy on the given test data and labels. In multi-label classification (which is what question 6 is doing), the subset accuracy is a harsh metric since each label needs to be correctly predicted. \n",
    "\n",
    "Later on in the jupyter notebook, there will be code showing on which class the model is consistently predicting incorrectly. This is called the confusion matrix. \n",
    "\n",
    "http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 align='Left'>50 Training Samples</h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 46.9 ms, sys: 1.43 ms, total: 48.4 ms\n",
      "Wall time: 50 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.57189999999999996"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_size = 50\n",
    "X_train = train_dataset[:sample_size].reshape(sample_size, 784)\n",
    "y_train = train_labels[:sample_size]\n",
    "%time regr.fit(X_train, y_train)\n",
    "regr.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 align='Left'>100 Training Samples</h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 111 ms, sys: 1.38 ms, total: 112 ms\n",
      "Wall time: 111 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.72970000000000002"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_size = 100\n",
    "X_train = train_dataset[:sample_size].reshape(sample_size, 784)\n",
    "y_train = train_labels[:sample_size]\n",
    "%time regr.fit(X_train, y_train)\n",
    "regr.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<h3 align='Left'>1000 Training Samples</h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.91 s, sys: 8.44 ms, total: 1.92 s\n",
      "Wall time: 1.92 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.83230000000000004"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_size = 1000\n",
    "X_train = train_dataset[:sample_size].reshape(sample_size, 784)\n",
    "y_train = train_labels[:sample_size]\n",
    "%time regr.fit(X_train, y_train)\n",
    "regr.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 align='Left'>5000 Training Samples plus Predicting Class Labels </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 18.2 s, sys: 53.5 ms, total: 18.3 s\n",
      "Wall time: 18.3 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.85150000000000003"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_size = 5000\n",
    "X_train = train_dataset[:sample_size].reshape(sample_size, 784)\n",
    "y_train = train_labels[:sample_size]\n",
    "%time regr.fit(X_train, y_train)\n",
    "regr.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# predict class labels for the test set\n",
    "predicted = regr.predict(X_test)\n",
    "predicted[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  9.10666599e-03,   1.46013123e-03,   4.28896213e-05,\n",
       "         8.16163108e-01,   1.53275425e-03,   1.67719413e-01,\n",
       "         4.46184298e-04,   6.29315654e-05,   1.40435337e-03,\n",
       "         2.06156808e-03])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regr.predict_proba(X_test)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 align='Left'>Confusion Matrix </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix, without normalization\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU4AAAEpCAYAAAD8osuLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuYXVV9//H3ZxJCuCUBMUlJIASDGKwKVG4iSQREEQ08\ntrYBtCC29ldUqDxaCIIEEaM+P4t4ob+iNk+q2BipltgixAi5gEAIJIIkxCgmhEAmAiq3NuTy/f2x\n18TDMHPm7Jmz5+w983nxnCf77LP22utMhm/WZa+1FBGYmVnj2lpdADOzqnHgNDPLyYHTzCwnB04z\ns5wcOM3McnLgNDPLyYFzEJI0XNKPJP1e0vf6kM/Zkm5tZtlaRdJbJa1pdTmsGuTnOMtL0tnAx4HX\nAc8Cq4DPRcRdfcz3/cBHgeNjEPwCSNoJTIqIR1tdFhsYXOMsKUkXA/8EfBYYDRwEfB14TxOynwD8\ncjAEzaTu95Q0pL8KYgNERPhVshcwAngOeG+dNMOALwObgMeBa4Hd0mdTgY3AxUB7SnNu+mwWsBV4\niawW+0HgSuDbNXlPAHYCben9ecCvU/pfA2el8+cCy2quewuwHPgdcC9ZjbbjszuAzwB3pnxuBfbr\n5rt1lP+TNeU/AzgNWAs8BcysSX808LN0303AV4Gh6bMl6bs8n+77vpr8/xF4EpjbcS5dcwjwNHBE\nen8AsAWY0urfDb/K8XKNs5yOB3YH/rNOmsuBY4A3Am9Kx5fXfD4W2Ifsf/q/Aa6XNDIiZgGfA+ZF\nxIiImJPSd66VBYCkPYHrgHdExAiy4Liqi3T7Av9FFsxfRRbI/zud73AWWbB9dfp+n6jz/caS/eNw\nAFlg/wZwDnAkMAW4QtKElHYH8A/AfmQ/u5OACwAiYmpK84b0fb9fk/8ospr8h2u/S2RN+n8EviNp\nD2AOMCciltYprw0iDpzl9CrgqYjYWSfN2cBVEfF0RDwNXAV8oObzl4CrI2JHRPyYrMZ1WC/LswN4\ng6ThEdEeEV0NopxO1vz/bkTsjIh5wCO8vGthTkT8OiK2AvOBI+rc8yWy/twdwDxgf+DLEfFiRKwG\nVpP9g0FEPBARyyPzGHADWQ2ylrr4TldGxLZUnpeJiG8BvyKrOY/h5f8o2SDnwFlOTwP7S6r393MA\n8FjN+w3p3K48OgXeF4G98xYkIl4E/gr4e+DJNBrfVQA+IJWh1gZgXM37zTnK83REdNSC/yf9uaXm\n8//puF7SoalcT0r6PXANWaCt57cRsa2HNN8EXg98tYG0Nog4cJbT3WT9kGfWSbOJrC+ywwTgiV7e\n7wVgz5r3f1L7YUT8JCJOJWveriWr0XX2BHBwp3MHpXIW7Z+BNcBrImIU8CleWcPsrKcBo73Iuh2+\nBcySNKoZBbWBwYGzhCLiWbJ+va9LOkPSHpKGSjpN0udTsnnA5ZL2l7Q/cAXw7V7echUwRdKBkkYC\nl3Z8IGm0pOmpr3MbWZO/qy6EW4BDJc2QNETSXwGTgR/1skx57AM8GxEvSnodWe241mayAZ88vgIs\nj4gPk323f+l7MW2gcOAsqYj4J7JR8cvJmqiPkQ14dAwYfRZYATwI/DwdX1Mvyzr3WgR8L+V1Hy8P\ndm2pHJvIRrOn8MrAREQ8A7ybbMDnqfTn6RHxu57u36AuB6+STwDnSHqWLMDN65R2FvBvkp6R9Bc9\n3UjSdOBU0gAT2fc/UtJZvSm4DTx+AN7MLCfXOM3McnLgNDPLyYHTzCynoa0uAIAkd7SaVVRE9PTo\nV8M0bESw7bk8l2yIiIObdf9GlWJwSFIMnzKr4fTb1t/Bbge/reH0T/3k070oVePy/NZcc/UsPnXF\nrKKKUrjelL/o37A8v8Ofu/oqLrviylz5b91ebwJX3+0+tPGGX2/Kv7PAv4CRewxpbuCUYvgRH2k4\n/f+u+npT79+oUtQ4zcx2qTthrhwcOM2sXNTvFcjcKhk420Yd3Ooi9NqJU6a1ugh9Uv3yd177o1qq\nXv6GuMZZjCGjJra6CL02Zeq0VhehT6pe/hNd/vJzjdPMLCfXOM3Mcmor/04mhYd2Se+U9IikX0q6\npOj7mVnFSY2/WqTQGmdaiPdrwMlk6zXeJ+nmiHikyPuaWYVVoKledAmPAdZFxIa0gvY8sk23zMy6\nNthrnGTbJmysef84WTA1M+taBWqcpRkc2rb+jl3HbaMOrvQjR2YD1bKli7lz6ZJib+LHkdhEtu9M\nh/F0swdNnrnnZtYaJ06Z9rJJEJ+/5jPNv4lrnNwHTEr7Xz8JzCDbW9vMrGuDPXBGxA5JHwUWkg1E\nfaubPbnNzDJtbqoTEbcCXe3DbWb2SoO9xmlmlpsHh8zMcnKN08wspwrUOMsf2s1scFFb46/uspA+\nLukXkh6UdKOkYZL2lbRQ0lpJt0kaWZN+pqR1ktZIOrWnIjpwmlm5tA1p/NUFSQcAHwOOiog3krWs\nzwIuBRZFxGHA7cDMlP5w4C+BycBpwPVS/WqvA6eZlUtz5qoPAfaSNBTYg2zizRnA3PT5XODMdDwd\nmBcR2yNiPbCOHqaGl6aP85lFxe1Eud9J+XYFzOt3dxQwe6LG/760o9D8hxT83FwFuqzqyrMLZW8U\nuQslwNAKPBf5Mn0cHIqIJyR9CXgMeBFYGBGLJI2JiPaUZrOk0emSccDdNVlsSue6VZrAaWYG1P2X\ndsfT69j59K96uFyjyGqXE4A/AN+XdA6v3Km61/9kOXCaWbnUqXEO2f8whuz/x/k0O351a1fJTgEe\njYhnACT9EHgL0N5R65Q0FtiS0m8CDqy5vts1NTq4j9PMyqXvo+qPAcdJGp4GeU4GVgMLgPNSmnOB\nm9PxAmBGGnmfCEwCltcromucZlYufewUj4jlkm4CVgLb0p83APsA8yWdD2wgG0knIlZLmk8WXLcB\nF0RE3Wa8A6eZlUsTZg5FxFXAVZ1OP0PWjO8q/WxgdqP5O3CaWblU4DEMB04zKxfPVTczy8k1TjOz\nfHqY7VgKDpxmVioOnGZmeZU/bjpwmlm5tLV5cMjMLBc31c3McnLgNDPLq/xx04HTzMrFNU4zs5wc\nOM3McnLgNDPLyYHTzCyv8sdNB04zKxfXOM3McnLgNDPLyYEzhyJ/WE//tPMK+s2179EfLTT/Z5Z/\ntdD8X9q+s9D8hxT8P8L2gjcmH77bkELz31lw+SsQh16uj+WV9Frge2Tb/wo4BLgC+HY6PwFYD/xl\nRPwhXTMTOB/YDlwUEQvr3aP8s+nNbFCR1PCrKxHxy4g4MiKOAv4MeAH4IXApsCgiDgNuB2am+x1O\ntnHbZOA04Hr1UJNz4DSzUulr4OzkFODXEbEROAOYm87PBc5Mx9OBeRGxPSLWA+uAY+plWpqmupkZ\nNH1Zub8CvpuOx0REO0BEbJY0Op0fB9xdc82mdK5bDpxmVi51KpJbn/gFLz3xcGPZSLuR1SYvSac6\ndyb3unPZgdPMSqVeE3z4uDcwfNwbdr1//v759bI6Dbg/Ip5K79sljYmIdkljgS3p/CbgwJrrxqdz\n3XIfp5mVShP7OM8C/r3m/QLgvHR8LnBzzfkZkoZJmghMApbXy9g1TjMrlWY8mihpT7KBoQ/XnP4C\nMF/S+cAGspF0ImK1pPnAamAbcEFE1G3GO3CaWbk04bnTiHgReHWnc8+QBdOu0s8GZjeaf6FNdUnj\nJd0u6WFJD0m6sMj7mVn1NflxpEIUXePcDlwcEask7Q3cL2lhRDxS8H3NrKIG/ZTLiNgMbE7Hz0ta\nQ/Z8lAOnmXVp0AfOWpIOBo4A7u2ve5pZ9ThwJqmZfhPZ5Pnnu0rz2c/M2nU8Zeo0pkyd1h9FM7Mc\nli5ZzNIli4u9SfnjJuph1L3vN5CGAv8F/DgirusmTfzPtuLKUfTqM6869mOF5l/51ZHavDpSPVVe\nHWnPYW1ERNPuICkmfvy/G07/m2tPb+r9G9UfNc5/BVZ3FzTNzGpVoale9ONIJwDnACdJWinpAUnv\nLPKeZlZtUuOvVil6VP0uoNh2jpkNKG0Fd+00g2cOmVmpVKGp7sBpZqVSgbjpwGlm5eKmuplZTq5x\nmpnl5D5OM7OcKhA3yxM4i5zBtLPg2VFFz+zZ79hiV+N7+p6vFJp/0T//oQX3iRU9s6fY3Isvf7O5\nxmlmlpMDp5lZThWIm96szczKpRkrwEsaKen7ktakHSiOlbSvpIWS1kq6TdLImvQzJa1L6U/tqYwO\nnGZWKk2aq34dcEtETAbeRLZ4+qXAoog4DLgdmJndT4eTbdw2mWxL4evVQ3+BA6eZlUpfa5ySRgAn\nRsQcgIjYHhF/AM4A5qZkc4Ez0/F0YF5Ktx5YBxxTr4wOnGZWKk2ocU4EnpI0J63IdkPaLnhMRLTD\nrm19Rqf044CNNddvSue65cEhMyuVeq3kZx9dxXO/WdVTFkOBo4CPRMQKSdeSNdM7P5fV6+e0HDjN\nrFTqzVUfNelIRk06ctf7J++Y21Wyx4GNEbEivf8PssDZLmlMRLRLGgtsSZ9vAg6suX58Otd9GXv4\nDmZm/aqvTfXUHN8o6bXp1MnAw8AC4Lx07lzg5nS8AJghaZikicAkYHm9MrrGaWal0qQH4C8EbpS0\nG/Ao8EGyRdXnSzof2EA2kk5ErJY0H1gNbAMuiB6mMjpwmlmpNCNuRsTPgaO7+OiUbtLPBmY3mr8D\np5mViqdcmpnlVIG46cBpZuXiGqeZWU4OnGZmOVUgbjpwmlm5uMZpZpZTBeKmA6eZlYtrnGZmOVUg\nbnYfOCXtV+/CiHim+cUxs8GurQKRs16N836yZZe6+hYBHFJIicxsUKu3OlJZdBs4I2JifxbEzAyg\nAnGz5z7OtPfGOcDEiLha0kHA2Iiou+xSXkVuvb294H2lt27fWWj+Re97vv/ZcwrN//G5f11o/sOG\nFrs6YtG7km/dtqPQ/Iv++TRbFQaHGvmJXg8cD5yd3j8HfL2wEpnZoNakzdoK1cio+rERcZSklQAR\n8TtJwwoul5kNUupyWKVcGgmc2yQNIbVYJL0aKLZtamaD1oDo4wS+AvwQGCPpGuAvgMsLLZWZDVpV\n6OPsMXBGxI2S7ifbtwPgzIhYU2yxzGywqkDcbHiztj3J9utoA/YorjhmNti1SQ2/uiNpvaSfS1op\naXk6t6+khZLWSrpN0sia9DMlrZO0RtKpPZaxpwSSPg3MBfYD9gfmSHJT3cwK0aRR9Z3AtIg4MiKO\nSecuBRZFxGHA7cDM7H46nGzjtsnAacD16qG/oJE+znOAN0XE/6abfB5YBXy2gWvNzHJpUh+neGXF\n8AxgajqeCywmC6bTgXkRsR1YL2kdcAxwb3eZN9JUfwIYXvN+d3rYrL0zSW2SHpC0IM91Zjb4NKnG\nGcBPJN0n6W/SuTFpz3UiYjMwOp0fB2ysuXZTOteteot8fDXd/A/Aw5J+kt6/nR42a+/CRWR7Fo/I\neZ2ZDTL1+i63rFnBlkdWNJLNCRHxZHp8cqGktbxyElivJ4XVa6p3lO5+sseROizOcwNJ44F3AdcA\nF+e51swGn3oVyTGT38yYyW/e9X71zTd0mS4inkx//lbSf5I1vdsljYmIdkljgS0p+SbgwJrLx9ND\nq7reIh9z612Yw7XAJ4GRPSU0M+trH6ekPYG2iHhe0l7AqcBVwALgPOALwLnAzemSBcCNkq4la6JP\noodWdSOLfBwKzAYOp6avMyJ6XFZO0ulAe0SskjSNOv+YXHP1rF3HJ06ZxpSp03rK3sz62bIli1m2\ndEmh9xjS96lDY4AfSgqyGHdjRCyUtAKYL+l8YAPZSDoRsVrSfLLuxG3ABRH1lx1SD58j6U7gSrKa\n43uAD5JF80/3VHpJnwPeD2wne/5zH+AHEfHXndLFC1uLm8X50o5iZ4juKHj1pT12G1Jo/l4dqb6i\nZ7JUeXWkfYYPISKa9gOSFO//zqqG03/n/Uc09f6NauQnukdE/JQsyG6IiFnA6Y1kHhGXRcRBqXY6\nA7i9c9A0M6slqeFXqzTyHOdWSW3AOkkfJes03bvYYpnZYDVQFvm4iGzK5YXA1cBJZB2ruUTEEqDY\nzhEzq7yBssjHfenwebL+TTOzwpQ/bNZ/AP5H1HlANCKmF1IiMxvUqr7L5f/tt1KYmSUViJt1H4B3\nf6SZ9bsB0cdpZtafKhA3HTjNrFyq3sdpZtbvKhA3yzOqXuQPa9iQYqfkRbEzIgu35cbzCs3/1cdd\nWGj+zyz/aqH59zAruc+KnjJaNVXv4/Soupn1uyr8M+JRdTMrlSasjlS4QpeVMzPLqwJxs6Fa8Rzg\nn8mWhnsb8G/Ad4oslJkNXlVYHanQZeXMzPJqU+OvVvGycmZWKhUYVG+oxlm7rNyfAR+gF8vKmZk1\nok1q+NWyMvaUICLui4jnI+LxiPhgRLw3Iu7pj8KZ2eDTluNVj6Q2SQ9IWpDe7ytpoaS1km6TNLIm\n7UxJ6yStkXRqT2VsZFT9Drp4ED4iTurpWjOzvJpYkbyIbAO2Een9pcCiiPiipEuAmcClkg4n27ht\nMtnWwIskHVpvw7ZG+jg/UXM8HPhzshF2M7Oma0YTXNJ44F3ANcDF6fQZwNR0PBdYTBZMpwPzImI7\nsF7SOrJ92O/tLv9GVoC/v9OpuyTV3XPYzKy3mlTjvBb4JDCy5tyYiGgHiIjNkkan8+OAu2vSbUrn\nutVIU32/mrdtZANEI7tJbmbWJ/UeM1r/83tZ/2C3FUEAJJ0OtEfEKknT6iTt9SoEjTTV7083EFkT\n/TfAh3p7QzOzeuo11Q854jgOOeK4Xe+XfOdrXSU7AZgu6V3AHsA+kr4NbJY0JiLaJY0FtqT0m4AD\na64fn851X8YGvsfkiDgkIiZGxKERcSpwX49XmZn1gtT4qysRcVlEHJSmhc8Abo+IDwA/As5Lyc4F\nbk7HC4AZkoZJmghMAup2RzYSOH/Wxbm7uzhnZtZnBc4c+jzwdklrgZPTeyJiNTCfbAT+FuCCeiPq\nUH89zrFkHaR7SDqSP+7aOYLsgXgzs6Yb0sTnkdIqb0vS8TPAKd2km022mFFD6vVxvoOsWjse+BJ/\nDJzPApc1egMzszyqsDpSvfU45wJzJf15RPxHP5bJzAaxKqwA30gf559JGtXxJk1b+myBZTKzQawK\nqyM1EjhPi4jfd7yJiN+RPZFvZtZ0fR1V7w+NPMc5RNLuEbEVQNIewO7FFsvMBquBsj3wjcBPJc1J\n7z9Itgq8mVnTVXpwqENEfEHSz/njMP7VEXFbscUys8GqAhXOhmqcRMStwK0Akt4q6esR8ZFmFmRn\ngXtXF/33sHXbjkLz3323YjduL7pp9Nt7vlJo/vuddGWh+T/106sKzb9oVRilrtVW+P+xfddQ4EwP\nwJ9Ftmbdb4AfFFkoMxu8qhDn680cei1ZsDwLeAr4HtmGbW/rp7KZ2SBU9T7OR4BlwLsj4lcAkj7e\nL6Uys0GrCqPq9Z7jfC/wJHCHpG9IOpniuwvNbJCrwnOc3QbOiPjPiJgBvA64A/gHYLSkf25kMyMz\ns94YKLtcvhAR342I95At+LESuKTwkpnZoFSFGmdDo+od0nTLG9LLzKzpmrmsXFEamaveJ5JGSvp+\n2q/4YUnHFn1PM6su5Xi1Sq4aZy9dB9wSEe+TNBQvgmxmdVRhVL3QwClpBHBiRJwHkPYtfrbIe5pZ\ntZU/bBbfVJ8IPCVpjqQHJN2QVlcyM+tSXweHJO0u6V5JKyU9JOnKdH5fSQslrZV0m6SRNdfMlLQu\ndSn2+NRQ0U31ocBRwEciYoWkLwOXAq+YXHzN1bN2HZ84ZRpTpk4ruGhmltfSJYtZtnRxoffo69z6\niNgq6W0R8aKkIcBdkn4M/DmwKCK+KOkSYCZwqaTDyaaTTyZ7cmiRpEPrbdhWdOB8HNgYESvS+5vo\n5lGmT10xq+CimFlfTZn68krN7M9+pun3aEYzOCJeTIe7k8W5AM4Apqbzc4HFZBW56cC81JW4XtI6\n4Bjg3iLL2K2IaAc2pnnvkG3JubrIe5pZtUlq+FUnjzZJK4HNwE8i4j5gTIpJRMRmYHRKPg7YWHP5\npnSuW/0xqn4hcKOk3YBHyRZCNjPrUr2G+sMrfsbDK+7uMY+I2AkcmQaofyjp9WS1zpcl620ZCw+c\nEfFz4Oii72NmA0O9muSfHn0Cf3r0Cbve3/Qv/1Q3r4h4VtJi4J1Au6QxEdEuaSywJSXbBBxYc9n4\ndK5bhT8Ab2aWR1uOV1ck7d8xYp6e4nk7sAZYAJyXkp0L3JyOFwAzJA2TNBGYBCyvV8b+aKqbmTWs\nCSvW/wkwV1JHfP1eRNwi6R5gvqTzgQ1kI+lExGpJ88nGX7YBF9QbUQcHTjMrmb6GzYh4iOwxyM7n\nn+GPe6d1/mw2MLvRezhwmlmpVGDGpQOnmZVLFVZHcuA0s1JRBWarO3CaWalUoMJZnsC5fcfOwvIe\nUvC2eXvuXuyPcWeRm84DO+sPIJbe0wXve/6qaZcVmv9v77im0PyL/v1ptgGzr7qZWX9xjdPMLCcH\nTjOznDw4ZGaWU8FDEk3hwGlmpeIap5lZTu7jNDPLyTVOM7Oc3MdpZpaTa5xmZjm5j9PMLKcKxE0H\nTjMrFy8rZ2aWV/njpjdrM7NyUY7/urxeGi/pdkkPS3pI0oXp/L6SFkpaK+m2jg3d0mczJa2TtEbS\nqT2V0YHTzEpFavzVje3AxRHxeuB44COSXgdcCiyKiMOA24GZ2f10ONnGbZOB04Dr1cOOcQ6cZlYq\nyvHqSkRsjohV6fh5sq2BxwNnAHNTsrnAmel4OjAvIrZHxHpgHXBMvTI6cJpZufQ1ctZmJR0MHAHc\nA4yJiHbIgiswOiUbB2ysuWxTOtctDw6ZWanUewD+/nuWcf89dzaWj7Q3cBNwUUQ8L6nzUvi9Xhrf\ngdPMSqVe7+Kbjz+RNx9/4q733/jK57vJQ0PJgua3I+LmdLpd0piIaJc0FtiSzm8CDqy5fHw61y03\n1c2sVJrUUv9XYHVEXFdzbgFwXjo+F7i55vwMScMkTQQmAcvrZe4ap5mVSx+f45R0AnAO8JCklWRN\n8suALwDzJZ0PbCAbSSciVkuaD6wGtgEXRNTfwdCB08xKpa+LfETEXcCQbj4+pZtrZgOzG72HA6eZ\nlUoFZlyWJ3DuNqS47tai/yJ6qNX3WdX3PS9a0T+fpxd/rtD8X3XylYXm/9gtVxSaf7NVIG6WJ3Ca\nmQGViJwOnGZWKm0VaKs7cJpZqZQ/bDpwmlnZVCByOnCaWal4zyEzs5wq0MXpwGlm5VKBuFn8XHVJ\nH5f0C0kPSrpR0rCi72lmFdbEZeWKUmjglHQA8DHgqIh4I1kNd0aR9zSzauvr1hn9oT+a6kOAvSTt\nBPYEnuiHe5pZRVWhj7PQGmdEPAF8CXiMbH2730fEoiLvaWbVVoGWerE1TkmjyPb5mAD8AbhJ0tkR\n8d3Oaa+5etau4xOnTGPK1GlFFs3MeuGuZUu4a9mSYm9SgRqnilygQtJfAO+IiL9N7z8AHBsRH+2U\nLl7YurPAchSWdb/YsdOLfLRS0VMAq7zIx+gRw4iIpv2AJMUvN7/YcPrXjt2zqfdvVNF9nI8Bx0ka\nDmwFTgbuK/ieZlZhVajoFBo4I2K5pJuAlWQrK68EbijynmZWbRWIm8WPqkfEVcBVRd/HzAaICkRO\nb9ZmZqXSJjX86o6kb0lql/Rgzbl9JS2UtFbSbZJG1nw2U9I6SWskndpjGfv8Lc3MmqhJjyPNAd7R\n6dylwKKIOAy4HZgJIOlwso3bJgOnAddL9XtaHTjNrFyaEDkj4k7gd51OnwHMTcdzgTPT8XRgXkRs\nj4j1wDrgmHpFdOA0s1IpcMrl6IhoB4iIzcDodH4csLEm3aZ0rlteHcnMSqVeI/nuO5dwz11Lm3Wr\nXj8g7cBpZqVSrx75lrdO5S1vnbrr/Ze/eE2erNsljYmIdkljgS3p/CbgwJp049O5brmpbmalIjX+\n6ikrXh6HFwDnpeNzgZtrzs+QNEzSRGASsLxexoOixln0tuRVnxBZ9JTCqv98ii7/k7cVO+XyT07+\nVKH5N1/ffx8lfReYBrxK0mPAlcDnge9LOh/YQDaSTkSsljQfWE02UeeC6GEu+qAInGZWHc34dzwi\nzu7mo1O6ST8bmN1o/g6cZlYqFZg45MBpZuUy6Bf5MDPLy9sDm5nlVf646cBpZuVSgbjpwGlm5VL0\n43HN4MBpZuVS/rjpwGlm5VKBuOnAaWblUoGWugOnmZWLH0cyM8upCjVOr45kZpaTa5xmVipVqHE6\ncJpZqbiP08wsJ9c4zcxyqkDcdOA0s5KpQOR04DSzUqlCH2clH0daumRxq4vQa1UuO7j8rXbn0sWt\nLkLhmrhZW2EqGTiXVfiXp8plB5e/1e5ctqTVRShcFQKnm+pmVipVaKo7cJpZqVThcST1sH1w/xRC\nan0hzKxXIqJpoU7SemBCjks2RMTBzbp/o0oROM3MqqSSg0NmZq3kwGlmlpMDp5lZTpUKnJLeKekR\nSb+UdEmry5OHpPGSbpf0sKSHJF3Y6jL1hqQ2SQ9IWtDqsuQlaaSk70tak/4ejm11mfKQ9HFJv5D0\noKQbJQ1rdZkGq8oETkltwNeAdwCvB86S9LrWliqX7cDFEfF64HjgIxUrf4eLgNWtLkQvXQfcEhGT\ngTcBa1pcnoZJOgD4GHBURLyR7FHCGa0t1eBVmcAJHAOsi4gNEbENmAec0eIyNSwiNkfEqnT8PNn/\ntONaW6p8JI0H3gV8s9VlyUvSCODEiJgDEBHbI+LZFhcrryHAXpKGAnsCT7S4PINWlQLnOGBjzfvH\nqVjg6SDpYOAI4N7WliS3a4FPAlV8hm0i8JSkOamr4QZJe7S6UI2KiCeALwGPAZuA30fEotaWavCq\nUuAcECTtDdwEXJRqnpUg6XSgPdWaRSUW/3qZocBRwNcj4ijgReDS1hapcZJGkbWwJgAHAHtLOru1\npRq8qhQ4NwEH1bwfn85VRmpi3QR8OyJubnV5cjoBmC7pUeDfgbdJ+rcWlymPx4GNEbEivb+JLJBW\nxSnAoxHxTETsAH4AvKXFZRq0qhQ47wMmSZqQRhNnAFUb2f1XYHVEXNfqguQVEZdFxEERcQjZz/72\niPjrVpfbRcsKAAADwElEQVSrURHRDmyU9Np06mSqNcj1GHCcpOGSRFb+ygxuDTSVWeQjInZI+iiw\nkCzgfysiKvOLI+kE4BzgIUkryfoJL4uIW1tbskHlQuBGSbsBjwIfbHF5GhYRyyXdBKwEtqU/b2ht\nqQYvz1U3M8upSk11M7NScOA0M8vJgdPMLCcHTjOznBw4zcxycuA0M8vJgXOAkbQjzcV+SNL3JA3v\nQ15TJf0oHb9H0j/WSTtS0t/34h5XSrq40fOd0syR9N4c95og6aG8ZTTrzIFz4HkhIo6KiDeQPSj9\nfzonSDNPGhUAEfGjiPhinXT7AhfkKmlr+MFl6zMHzoFtGX+cpvqIpLmpxjVe0tsl/UzSilQz3RN2\nLRa9RtIKYFdtTtK5kr6ajkdL+oGkVZJWSjoOmA28JtV2v5DSfULS8pTuypq8PiVpraSlwGE9fQlJ\nf5PyWZkWIq6tRb9d0n3p+52e0rdJ+qKke9O9/7bPP0mzGg6cA49g14IipwEdTdNDga+lmuiLwOXA\nyRHxZuB+4GJJu5NN4zs9nR/bKe+O2tpXgMURcQTZQhkPk6009KtU271E0tuBQyPiGOBI4M2S3irp\nKOAvgTcCpwNHN/Cd/iMijomII4FHgA/VfDYhIo4G3g38v7SOwYfIll07lmwd1w9LyrPlrFldlZmr\nbg3bQ9ID6XgZ8C2ydUvXR8R96fxxwOHAXanZvhtwN/A6shV4Hk3pvgN0VVs7CfgAQGRzdp+TtF+n\nNKeS1QYfIAvme5EF7xHADyNiK7C1wS043ijpamBUyue2ms/mp3L8StKv03c4FXiDpPelNCPSvdc1\ncC+zHjlwDjwvpvUmd0ldmi/UngIWRsQ5ndK9icbW2Wykn1DA7Ij4Rqd7XNTAtZ3NAaZHxC8knQtM\n7aYsSu8FfCwiftLp3q51WlO4qT7wdBf4as/fA5wg6TUAkvaUdChZM3iCpIkp3Vnd5PVT0kBQ6k8c\nATwH7FOT5jbgfEl7pXQHSHo1sBQ4U9LukvYB3tPAd9ob2JxWNTqn02fvU+Y1ZKu8r033viB1VyDp\n0JrV3qu2ALOVkGucA093tcFd5yPiKUnnAf+e+jUDuDwi1kn6O+AWSS+QNfX37iKvfwBukPQhsk3o\n/j4i7k2DTQ8CP079nJOBu1ON9zng/RGxUtJ84EGgHVjewHf6dEq3hWy7kdoA/Vj6bB/g7yLiJUnf\nBA4GHkhdEVuAM3v4+Zg1zMvKmZnl5Ka6mVlODpxmZjk5cJqZ5eTAaWaWkwOnmVlODpxmZjk5cJqZ\n5fT/Ad7JA6WPwTBiAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x107c5f290>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normalized confusion matrix\n",
      "[[ 0.84  0.01  0.01  0.01  0.01  0.01  0.02  0.05  0.02  0.03]\n",
      " [ 0.01  0.84  0.    0.04  0.02  0.02  0.01  0.02  0.03  0.01]\n",
      " [ 0.    0.01  0.89  0.01  0.03  0.01  0.04  0.    0.01  0.01]\n",
      " [ 0.01  0.02  0.01  0.88  0.01  0.02  0.02  0.01  0.01  0.02]\n",
      " [ 0.01  0.03  0.05  0.01  0.77  0.04  0.01  0.01  0.05  0.01]\n",
      " [ 0.01  0.01  0.01  0.01  0.02  0.89  0.01  0.01  0.02  0.02]\n",
      " [ 0.01  0.01  0.05  0.01  0.01  0.02  0.85  0.01  0.02  0.01]\n",
      " [ 0.03  0.01  0.    0.01  0.02  0.02  0.01  0.86  0.02  0.02]\n",
      " [ 0.01  0.01  0.01  0.01  0.01  0.02  0.02  0.01  0.84  0.06]\n",
      " [ 0.01  0.01  0.01  0.02  0.01  0.01  0.01  0.01  0.04  0.86]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUsAAAEpCAYAAAAaiwDPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xe8HVW5//HP96QAAUJABCExoRcVRC4EEEgCWKJI+Vmu\nAUREL3K9ICgXRVGk267doJco8gNBIsYSbBCFNHroLTFBIKRQpPeQ8tw/ZhImm11mn7Pn7Nk53zev\n/WLK2muevc85T9ZaM7NGEYGZmdXX1e4AzMw6gZOlmVkOTpZmZjk4WZqZ5eBkaWaWg5OlmVkOTpYd\nRNLpkn6ZLr9Z0nOS1OJjPChp/1bWmeOYn5H0aPp5NuxBPc9L2qJ1kbWPpHskjWp3HPYaJ8sMSQ9J\nekzSOpltn5I0tZ1xVQiAiFgQEYOjwy+UldQf+C7wrvTzPN3duiJi/Yh4qGXBFUDShZLOalQuIt4W\nETN6IybLx8lydUHynXyuyvamtbrVt4Z6E7AWMLvdgZSBpH7tjsGqc7J8vf8B/lvS4Go7Jb1T0s2S\nnpZ0k6S9MvumSjpH0rWSXgS2TLedLem6tJs4WdJGki6R9Gxax/BMHT+Q9HC6b5akfWrEMULSCkld\nkvZM634ufb0s6YG0nCR9SdL9kv4laaKkIZl6jkxb1P+SdGq9L0bS2pK+m5Z/WtIMSWul+w5Ou45P\nSbpG0g6Z9z0o6b8l3Zm+7zJJAyVtC8xJiz0t6e/Zz1XxvX4yXd5a0jRJz0h6XNJlmXIrJG2VLg+W\ndHFa5kFJX8mUO0rSTEn/k8b7T0lj63zuByWdnMb/vKSfSdpE0l/S73uKpA0y5S+X9Ej6WadJ2jHd\nfgxwBPDF9H2TM/V/UdKdwAuS+ikzHCLpz5K+k6l/oqSf1/tZWQEiwq/0BTwI7A9MAs5Ot30KuCZd\n3hB4Cjic5B+acen6hun+qcBDwA7p/v7ptrnAFsD6wL0kCWK/tMxFwAWZGA4HhqT7Pg88AgxM950O\nXJwujwCWA10Vn6E/MA04J10/Ebge2AwYAPwU+FW67y3A88De6b7vAq8C+9f4fs4DriFpDQrYM33f\ndsAL6XfXD/gCMA/on/lebwQ2TT/bfcCnKz6Han2u9Dv8ZLr8K+DL6fJA4J2ZcsuBrdLli4HfA4PS\nOv8BHJ3uOwpYAnwy/Rz/CSxq8HtxPbBx+j0+BtwC7JzGcDVwWqb8J9LjDgC+B9ye2XchcFaV+m8D\nNgfWyv4upsubAo8CY0iS7f3AoHb/vfS1V9sDKNOL15LlW4GngTewerL8GHBjxXuuBz6eLk8FzqjY\nP3XlH3e6/h3gz5n1DwC31YnpKWCndDlPsvwpcEVm/T5gv8z6ZiQJsQs4jTRxpvsGpUnkdckyTSov\nAW+rsu+rwMSKsguBUZnv9bDM/m8BP6n2Oap9LlZPlhcB/wsMrRLHCmCr9LMtAbbP7Pt05ud4FDA3\ns2+d9Jib1Pm9yMY/CTgvs3488Lsa7x2SxrV+ul4rWR5V7Xcxs/7/gIeBx4G92v230hdf7oZXERH3\nAn8Cvlyxa3NgfsW2+cDQzPqCKlU+lll+ucr6eitX0u7efWkX7mlgMEmLpiFJxwKjSFqnK40Afp92\nN58iSZ5LSVorm2fjjYiXgCdrVL8xydjiA1X2rfa9RPLXvYDVv5fsZ36JzGdu0hdIkuHNku6WdHSN\nWPuTJJeVKn9Oj2bifZkkwdeLKdfPMB0W+WY67PEMSdILGv8MFzbY/yeSVvs/IuKGBmWtAE6WtZ0B\nHMPqf2CLSbrTWcOBRZn1bp+dlrQvSTL4cERsGBEbAs+R/CHnee+ZwMER8UJm18PA+yJio/S1YUSs\nGxGPkHTx35ypYxBJa7qaJ4BXgK2r7FtMkpSz3kzjBFDNi+n/B2W2vWnlQkQ8HhGfjoihJN3nn6wc\np6yIdWlFTCNY/edUlCOAg0hahUNIfl/Eaz/DWr8fjX5vvk7yD91mksa1IE5rkpNlDRHxT+DXwAmZ\nzX8BtpU0Lh2E/yiwI/DHFh12PZI/8ifTEyBfIxnnrEWQXHOZxvrxNO6s84GvrzyJJOmNkg5O900C\nPqDkpNUA4CxqJOa0tfgL4HuSNsucWBoAXA4cKGk/Sf0lnUySWPO2gFYdMyKeIElqH0uP8UkyCVrS\nhyWt/AfsGZIu7oqKWFekMZ0raT1JI0jGf3+ZM56eWI9kCOBpSesC32D1RPgYyVBBbkqutzwKOJJk\nPPTHkjZrSbSWm5Pl6ir/dT+LpIWz8trGp0jGGE8mab2cDBwYr10bWK110ExL86r0NZek+/YS1bv1\nlXXvD2wCTErPsj4v6e503w+BycAUSc+SjLGOTD/PfcBxwGUkrcMnqd8aPBm4G5iVlv0mydjiXJLx\n3PHAv4ADgYMiYllFnI0+x0rHAF8k+Y53BK7L7NsduEnSc8AfgBPitWsrs/WcQPL9PQDMAC6JiAub\niKHevnplLyZpzS8C7iH5vrMuAN6aDov8rk59yeCvtD7JOO1xEfFoRFwL/Jxk7NN60cozkGZmVodb\nlmZmOThZmpnl4GRpZpZD/3YHACDJA6dmHSoiWjYHggYODpY+38xb5kfEFq06fj2lOMEjKdYedUbu\n8ksfmsqALfbLXf6Jv32tG1Hl18xvyrlnn8FXTjujqFAK1534i/4Na+Z3+Otnn8mpp53eVP1Llq1o\nXKgH1uqfv4PXnfhXFPgD2GCdfq1NllKsvctxucu/csd5LT1+PaVoWZqZraJyjg46WZpZuZR0ZsOO\nTJZdQ7Zodwjdtu+oMe0OoUc6P/7R7Q6hRzo9/lzcsmydfkO2bHcI3TZq9Jh2h9AjnR7/vo6//Nyy\nNDPLwS1LM7Mcusr5ZI3CU7iksZLmSJor6ZSij2dmHU7K/+pFhbYs0+eojAcOIJnVZpakyRExp/47\nzazPKmk3vOioRgLzImJ+RCwFJgKHFHxMM+tkfbFlSTLLeHY+xoWkcymamVVV0pZlaU7wLH1o6qrl\nriFbdPTlQWZrqpkzpnHtjOnFHqSPXjq0iOQZNSsNo8ZzUJq519vM2mPfUWNWuzHhm+ee1fqD9NGW\n5Sxgm/QZKI+QPGf7sIKPaWadrC8my4hYLul4YArJyaQLImJ2kcc0sw7X1Te74UTElcD2RR/HzNYQ\nfbFlaWbWtD56gsfMrDluWZqZ5VDSlmU5U7iZ9V3qyv+qVUWDOSkkDZZ0haQ7JN0t6RONwnLL0szK\npYezDuWck+I44N6IOFjSxsA/JF0SEctqhtWjqMzMWq3n94bnmZMigPXT5fWBJ+slSihRy/Kpvxf3\nBMaN9m/uaXjNenpqAXcxZLzy6vJC6+9X8HVtJR2Cyq2Zpy92R5FPXwToX9LrFmvq+QmePHNSjAeu\nkLQYWA/4aKNKS5MszcyAuv+6Ln9yHiuevL8VR3kvcHtE7C9pa+BvknaOiBdqvcHJ0szKpU7Lst/G\n29Nv49fucVl+/5XViuWZk+Jo4BsAEfFPSQ8COwC31Dq2xyzNrFx6fjZ81ZwUkgaSzElxRUWZ+cC7\nACRtCmwHPFAvLLcszaxcejjIXWtOCknHJrtjAnAO8P8l3ZW+7YsR8VS9ep0szaxcWnAHT7U5KSLi\n/MzyIyTjlrk5WZpZuZT08gknSzMrF98bbmaWg1uWZmaNycnSzKwxJ0szszzKmSudLM2sXLq6fILH\nzKwhd8PNzHJwsjQzy6OcudLJ0szKxS1LM7McnCzNzHJwsjQzy8HJ0swsj3LmSidLMysXtyzNzHJw\nsjQzy8HJsoEiv6Anrz6zsLoBNtz9+ELrf+rmHxda/6vLVhRaf7+Cf/mXFfzg7bUH9Cu0/hUFx1/S\n3FNbC+KVNBb4Aa89g+dbFftPBo4AAhgA7AhsHBHP1KqznHesm1mfJSn3q8b7u4DxJM/YeStwmKQd\nsmUi4jsR8Y6I2BX4MjCtXqIEJ0szK5meJktgJDAvIuZHxFJgInBInUMeBlzWKK7SdMPNzKAlU7QN\nBRZk1heSJNDXkbQOMBY4rlGlTpZmVi51xiyXLL6HVxff28qjHQRc26gLDk6WZlYy9U72rj10J9Ye\nutOq9RduvbxasUXA8Mz6sHRbNePI0QUHj1maWcm0YMxyFrCNpBGSBpIkxCuqHGcDYDQwOU9cblma\nWan09DLCiFgu6XhgCq9dOjRb0rHJ7piQFj0UuCoiXs5Tr5OlmZVLC66zjIgrge0rtp1fsX4RcFHe\nOgvthksaJukaSfdKulvSCUUez8w6Xwu64YUoumW5DDgpIu6QtB5wq6QpETGn4OOaWYfqk7c7RsSj\nwKPp8guSZpNcA+VkaWZV9clkmSVpC2AX4KbeOqaZdZ4+nSzTLvgk4MSIeKFamXPOOmPV8qjRYxg1\nekxvhGZmTZgxfRozpk8r9iDlzJUoougZT9Qf+BPw14j4YY0y8fLS4uIoelaXN+zx2ULr7/hZh7o8\n61A9nTzr0KCBXUREy44gKbb8/J9zl3/w+we29Pj19EbL8hfAfbUSpZlZVlm74UVfOrQ3yZxx+0u6\nXdJt6TxzZmZVSflfvanos+HXAcX2YcxsjdJV8LBNd/kOHjMrlbJ2w50szaxUSpornSzNrFzcDTcz\ny8EtSzOzHDxmaWaWQ0lzZXmSZZF3Eq0o+C6lou+w2WiPYme2e/LGHxVaf9Hff/+Cx7iKvsOm2NqL\nj7/V3LI0M8vBydLMLIeS5ko/sMzMyqUVM6VLGitpjqS5kk6pUWZMehv2PZKmNorLLUszK5Wetiwl\ndQHjgQOAxcAsSZOzT2hIn+x4HvCeiFgkaeNG9bplaWal0oKW5UhgXkTMj4ilwETgkIoyhwO/jYhF\nABHxRKO4nCzNrFRaMOvQUGBBZn1hui1rO2AjSVMlzZJ0ZKO43A03s1KpNxb53AN38PyDd7TiMP2B\nXYH9gXWBGyTdEBH313uDmVlp1Ls3fMg272DINu9Ytf7I1KqP/V4EDM+sD0u3ZS0EnoiIV4BXJM0A\n3g7UTJbuhptZqbSgGz4L2EbSCEkDgXHAFRVlJgP7SOonaRCwBzC7XlxuWZpZqfT0ovSIWC7peGAK\nSYPwgoiYLenYZHdMiIg5kq4C7gKWAxMi4r569TpZmlmptOKi9Ii4Eti+Ytv5FevfAb6Tt04nSzMr\nFd/uaGaWQ0lzpZOlmZWLW5ZmZjk4WZqZ5VDSXOlkaWbl4palmVkOJc2VTpZmVi5uWZqZ5VDSXFk7\nWUraqN4bI+Kp1odjZn1dV0mzZb2W5a0kD56rFnkAWxUSkZn1afVmHWqnmskyIrbszUDMzABKmisb\nj1kqGW09AtgyIs6WNBx4U0Tc3MpAiny09LKCn5u8ZNmKQusv+rneGx9+YaH1L7zo44XWP7B/sTMN\nFv3U7SVLlxdaf9HfT6uV9QRPnm/xJ8BeJM+sAHie5EE/ZmYt14L5LAuR52z4HhGxq6TbASLi6XRC\nTTOzllPV0yTtlydZLpXUj7Q3IumNQLH9TjPrszp2zBL4EfB7YFNJ5wIfBr5aaFRm1meVdcyyYbKM\niEsl3UrywHKAQyOi7rMqzMy6q6S5MvcDywYB/dLy6xQXjpn1dV1S7lctksZKmiNprqRTquwfLekZ\nSbelr4a95TyXDn0N+AjwW5IL1C+U9JuIOKfRe83MmtXTlqWkLmA8SW94MTBL0uSImFNRdEZEHJy3\n3jxjlkcAb0+fr4ukbwJ3AE6WZtZyLRizHAnMi4j5aX0TgUOAymTZ1IHydMMXA2tn1tfi9Q8sr0tS\nV9rUrXx2r5nZalpwneVQYEFmfWG6rdJeku6Q9GdJb2kUV72JNH5McrnQs8C9kv6Wrr8baPbunROB\n+4DBTb7PzPqYemORj8++hcfn3NKKw9wKDI+IlyS9D/gDsF29N9Trhq+M6FaSS4dWmtZMRJKGAe8H\nzgVOaua9Ztb31Osbb7rjbmy6426r1u+bPKFasUXA8Mz6MCp6wxHxQmb5r5J+ImmjerOp1ZtI46I6\nMTfj+8AXgA1aVJ+ZrcFaMGY5C9hG0gjgEWAccFjFMTaNiMfS5ZGAGk07meds+LbAN4C3kBm7jIiG\nU7RJOhB4LCLukDSGOv9onHv2GauW9x01hlGjxzSq3sx62czp05g5Y3qhx+jXw1t4ImK5pOOBKSTn\nZS6IiNmSjk12xwTgw5I+AywFXgY+2qheRYPpfiRdC5xO0kI8CDga6IqIrzWsXPo68DFgGcn1mesD\nv4uIj1eUixeXFHcH5avLi707c3nBsxqtM6BfofV71qH6ir6jpJNnHVp/7X5ERMu+IEnxsUvuyF3+\nko/t0tLj15PnW1wnIq4mSazzI+IM4MA8lUfEqRExPG2FjgOuqUyUZmZZknK/elOe6yyXpBd5zkub\ntouA9YoNy8z6qk6eSONEktsdTwDOBvYHjmr2QBExHSh2sMPMOl4nT6QxK118gWS80sysMOVMlfUv\nSv8jdWbUb+aeSjOzvDrx6Y7f6bUozMxSJc2VdS9K9/iimfW6jh2zNDPrTSXNlU6WZlYunThmaWbW\n60qaK8tzNrzIL2hgv2Jvh4ti70Ys3OOXfqLQ+t+45wmF1v/UzT8utP4GdwT3WNG3a3aaThyz9Nlw\nM+t1Zf2nw2fDzaxUejrrUFEKnaLNzKxZJc2VuVq8FwI/JZlmbT/gYuCSIoMys76rrLMOFTpFm5lZ\ns7qU/9WbPEWbmZVKSU+G52pZZqdo+zfgSLoxRZuZWR5dUu5Xr8bVqEBEzIqIFyJiYUQcHREfjIgb\neyM4M+t7upp41SJprKQ5kuZKOqVOud0lLZX0wUZx5TkbPpUqF6dHxP6N3mtm1qyeNhjTYcPxwAHA\nYmCWpMkRMadKuW8CV+WpN8+Y5cmZ5bWBD5GcGTcza7kWdK9HAvMiYj6ApInAIcCcinKfBSYBu+ep\nNM9M6bdWbLpO0s15Kjcza1YLhiKHAgsy6wtJEmjmGNocODQi9kufG95Qnm74RpnVLpKTPBvkqdzM\nrFn1Lgl66M6beOium1pxmB8A2bHMhik6Tzf8VpIxS5F0vx8EPtWd6MzMGqnXDd9qlz3Zapc9V61P\nv2R8tWKLgOGZ9WHptqzdgIlKrmzfGHifpKURcUWtY+dJljtGxCvZDZLWyvE+M7OmtaAbPgvYRtII\n4BFgHHBYtkD2dm1JFwJ/rJcoId91ltdX2XZDjveZmTWtp3fwRMRy4HhgCnAvMDEiZks6VtKnq70l\nT1z15rN8E8lA6TqS3sFrffrBJBepm5m1XL8WNC0j4kpg+4pt59co+8k8ddbrhr8X+ARJf/+7vJYs\nnwNOzVO5mVmzyjrrUL35LC8CLpL0oYj4bS/GZGZ9WFlnSs8zZvlvkoasXJG0oaRzCozJzPqwss46\nlCdZvi8inlm5EhFPA+8vLiQz68uk/K/elOfSoX6S1oqIJQCS1gF86ZCZFaKTH4V7KXB1ei0SwNEk\ns6WbmbVcx53gWSkiviXpTuBd6aazIyLXLB1mZs0qacMyV8ty5TVLVwJI2kfSeRFxXCsDWVHgs5mL\n/u6XLF1eaP1rDSj2weRFd3v+deOPCq1/o/1PL7T+J64+s9D6i1bWs8u1dBX+F9s9uZJlelH6YcC/\nk9wb/rsigzKzvqusub3eHTzbkSTIw4AngF+TPLRsv16Kzcz6oE4cs5wDzAQ+EBH3A0j6fK9EZWZ9\nVlnPhte7zvKDJDN2TJX0M0kHUPzwn5n1cWW9zrJmsoyIP0TEOGAHYCrwOWATST+V9J7eCtDM+pZO\nfrrjixHxq4g4iGRSjdtZfYZhM7OWKWvLMtfZ8JXSWx0npC8zs5ZrxRRtRchzb3iPSNpA0m8kzZZ0\nr6Q9ij6mmXUuNfHqTU21LLvph8BfIuIjkvrjiYPNrI6yng0vNFlKGgzsGxGfAIiIZSSTB5uZVVXO\nVFl8N3xL4AlJF0q6TdKEdNYiM7OqWnGCR9JYSXMkzZX0uhPSkg6WdKek2yXdLGnvRnEV3Q3vD+wK\nHBcRt0j6AfAl4HU385579hmrlvcdNYZRo8cUHJqZNWvG9GnMnDGt0GP09F52SV3AeOAAYDEwS9Lk\niJiTKfb3lU9zlLQTcDmwY716i06WC4EFEXFLuj6JGpcdfeW0MwoOxcx6atTo1Rsy3zjnrJYfowXd\n3ZHAvIiYDyBpInAIyV2JAETES5ny6wEreiGu2iLiMWBBep85JJn+viKPaWadTVLuVw1DgQWZ9YXp\ntsrjHCppNvBHoOETHnvjbPgJwKWSBgAPkEwebGZWVb1O+L23XM+9t9zQkuNExB+AP0jaBzgHeHe9\n8oUny4i4E9i96OOY2Zqh3pjl23bfm7ft/tq5mEnnf69asUXA8Mz6sHRbVRFxraStJG0UEU/VKlf4\nRelmZs3oauJVwyxgG0kjJA0ExgFXZAtI2jqzvCswsF6ihN7phpuZ5dbTs+ERsVzS8cAUkpx6QUTM\nlnRssjsmAB+S9HHgVeBlkonN63KyNLNSacVF6emjcLav2HZ+ZvnbwLebqdPJ0sxKpaR3OzpZmlm5\nlHXWISdLMysVlfTucCdLMyuVkjYsy5Msly1veLdRt/Ur+HFxg9Yq9mtcUeRD1YEVUWz9RXuy4Od6\nv2HMqYXW/6+p5xZaf9G/P63W0c8NNzPrLW5Zmpnl4GRpZpaDT/CYmeVQ8CmGbnOyNLNSccvSzCwH\nj1mameXglqWZWQ4eszQzy8EtSzOzHDxmaWaWQ0lzpZOlmZWLp2gzM8ujnLnSDywzs3JRE//VrEMa\nK2mOpLmSTqmy/3BJd6avayXt1CgutyzNrFR62guX1AWMBw4AFgOzJE2OiDmZYg8AoyLiWUljgZ8B\ne9ar1y1LMysVNfGqYSQwLyLmR8RSYCJwSLZARNwYEc+mqzcCQxvF5WRpZuXS82w5FFiQWV9I/WT4\nH8BfG4XlbriZlUq9schbb5zJrTde27pjSfsBRwP7NCrrZGlmpVJvzHK3vfZlt732XbX+sx99s1qx\nRcDwzPqwdFvFcbQzMAEYGxFPN4rL3XAzK5UWjFnOAraRNELSQGAccMVqx5CGA78FjoyIf+aJyy1L\nMyuXHp4Nj4jlko4HppA0CC+IiNmSjk12xwTgNGAj4CeSBCyNiJH16nWyNLNSacVEGhFxJbB9xbbz\nM8vHAMc0U6eTpZmVSknvdixPshzQr7jh06K//Cj4udud/lzvohX9/Tw57euF1v+GA04vtP6H/3Ja\nofW3WklzZXmSpZkZUNps6WRpZqXSVdJ+uJOlmZVKOVOlk6WZlU1Js6WTpZmVip/BY2aWQ0mHLJ0s\nzaxcSpori783XNLnJd0j6S5Jl6b3apqZVdeCm8OLUGiylLQ58Flg14jYmaQlO67IY5pZZ2vFYyWK\n0Bvd8H7AupJWAINIpnk3M6uqrGOWhbYsI2Ix8F3gYZL55J6JiL8XeUwz62wl7YUX27KUNITk2Rcj\ngGeBSZIOj4hfVZY99+wzVi3vO2oMo0aPKTI0M+uG62ZO57qZ04s9SElblipyEghJHwbem06HhKQj\ngT0i4viKcvHikhUFxlFY1b1i+QpPpNFORd9+18kTaWwyeCAR0bIvSFLMffSl3OW3e9Oglh6/nqLH\nLB8G9pS0NrCE5NGUswo+ppl1sLI2bgpNlhFxs6RJwO3A0vT/E4o8ppl1tpLmyuLPhkfEmcCZRR/H\nzNYQJc2WfmCZmZVKl5T7VYuksZLmSJor6ZQq+7eXdL2kVySdlCcu3+5oZqXS04alpC5gPMk5ksXA\nLEmTI2JOptiTJDfMHJq3Xrcszaxcen6h5UhgXkTMj4ilwESSSxhXiYgnIuJWYFnesJwszaxUWnC7\n41BgQWZ9YbqtR9wNN7NSqXfp0A3XTufG62b0XjAZTpZmVir1xizfuc9o3rnP6FXrP/j2udWKLQKG\nZ9aHpdt6xN1wMysVKf+rhlnANpJGpFNCjgOuqHfIPHH1iZZl0Y/d7vSbEYu+na/Tv5+i43/kqmJv\nd9zsgK8UWn/r9ez3MSKWSzoemELSILwgImZLOjbZHRMkbQrcAqwPrJB0IvCWiHihVr19IlmaWedo\nxb/dEXElsH3FtvMzy48Bb26mTidLMyuVkt7A42RpZuXSJyfSMDNrlh+Fa2aWRzlzpZOlmZVLSXOl\nk6WZlUvRl7J1l5OlmZVLOXOlk6WZlUtJc6WTpZmVS0l74U6WZlYuvnTIzCyHsrYsPeuQmVkOblma\nWamUtWXpZGlmpeIxSzOzHNyyNDPLoaS50snSzEqmpNnSydLMSqWsY5YdeenQjOnT2h1Ct3Vy7OD4\n2+3aGdPaHULhWvDAskJ0ZLKc2cG/MJ0cOzj+drt25vR2h1C4siZLd8PNrFTK2g13sjSzUinrpUOK\noh+qnScIqf1BmFm3RETL0pukh4ARTbxlfkRs0arj11OKZGlmVnYdeYLHzKy3OVmameXgZGlmlkNH\nJUtJYyXNkTRX0intjqcZkoZJukbSvZLulnRCu2PqDkldkm6TdEW7Y2mWpA0k/UbS7PTnsEe7Y2qG\npM9LukfSXZIulTSw3TH1JR2TLCV1AeOB9wJvBQ6TtEN7o2rKMuCkiHgrsBdwXIfFv9KJwH3tDqKb\nfgj8JSJ2BN4OzG5zPLlJ2hz4LLBrROxMctnfuPZG1bd0TLIERgLzImJ+RCwFJgKHtDmm3CLi0Yi4\nI11+geQPdWh7o2qOpGHA+4GftzuWZkkaDOwbERcCRMSyiHiuzWE1qx+wrqT+wCBgcZvj6VM6KVkO\nBRZk1hfSYclmJUlbALsAN7U3kqZ9H/gC0InXm20JPCHpwnQYYYKkddodVF4RsRj4LvAwsAh4JiL+\n3t6o+pZOSpZrBEnrAZOAE9MWZkeQdCDwWNo6FqWdSKum/sCuwHkRsSvwEvCl9oaUn6QhJD2pEcDm\nwHqSDm9vVH1LJyXLRcDwzPqwdFvHSLtPk4BfRsTkdsfTpL2BgyU9AFwG7Cfp4jbH1IyFwIKIuCVd\nn0SSPDvFu4AHIuKpiFgO/A54Z5tj6lM6KVnOAraRNCI9CzgO6LQzsr8A7ouIH7Y7kGZFxKkRMTwi\ntiL57q+JiI+3O668IuIxYIGk7dJNB9BZJ6oeBvaUtLYkkcTfMSeo1gQdM5FGRCyXdDwwhSTJXxAR\nHfPLImlKWJ0jAAADj0lEQVRv4Ajgbkm3k4z7nRoRV7Y3sj7lBOBSSQOAB4Cj2xxPbhFxs6RJwO3A\n0vT/E9obVd/ie8PNzHLopG64mVnbOFmameXgZGlmloOTpZlZDk6WZmY5OFmameXgZLmGkbQ8vff5\nbkm/lrR2D+oaLemP6fJBkr5Yp+wGkj7TjWOcLumkvNsrylwo6YNNHGuEpLubjdEMnCzXRC9GxK4R\nsRPJxcv/WVkgvQMkrwCIiD9GxLfrlNsQ+K+mIm0PX1hs3eJkuWabyWu3iM6RdFHashom6d2Srpd0\nS9oCHQSrJlieLekWYFWrTdJRkn6cLm8i6XeS7pB0u6Q9gW8AW6et2m+l5U6WdHNa7vRMXV+R9A9J\nM4DtG30ISf+R1nN7OnlvtrX8bkmz0s93YFq+S9K3Jd2UHvuYHn+T1uc5Wa55BKsm7XgfsLLbuS0w\nPm1xvgR8FTggInYDbgVOkrQWyS10B6bb31RR98pW2Y+AaRGxC8lkFPeSzOBzf9qqPUXSu4FtI2Ik\n8A5gN0n7SNoV+HdgZ+BAYPccn+m3ETEyIt4BzAE+ldk3IiJ2Bz4A/G86b8CnSKYw24NkHtRPS2rm\n8apmr9Mx94ZbbutIui1dnglcQDLv50MRMSvdvifwFuC6tEs+ALgB2IFkZpsH0nKXANVaZfsDRwJE\ncr/s85I2qijzHpJW320kCXxdkoQ9GPh9RCwBluR8PMXOks4GhqT1XJXZd3kax/2S/pl+hvcAO0n6\nSFpmcHrseTmOZVaVk+Wa56V0vsZV0iHKF7ObgCkRcURFubeTb57KPON+Ar4RET+rOMaJOd5b6ULg\n4Ii4R9JRwOgasShdF/DZiPhbxbHdurRuczd8zVMr2WW33wjsLWlrAEmDJG1L0sUdIWnLtNxhNeq6\nmvRkTjo+OBh4Hlg/U+Yq4JOS1k3LbS7pjcAM4FBJa0laHzgox2daD3g0nS3oiIp9H1Fia5LZ0P+R\nHvu/0qEIJG2bmRW90yYttpJwy3LNU6vVt2p7RDwh6RPAZek4ZQBfjYh5ko4F/iLpRZJu/HpV6voc\nMEHSp0gexPaZiLgpPWF0F/DXdNxyR+CGtGX7PPCxiLhd0uXAXcBjwM05PtPX0nKPkzyKI5uUH073\nrQ8cGxGvSvo5sAVwWzrM8DhwaIPvx6wuT9FmZpaDu+FmZjk4WZqZ5eBkaWaWg5OlmVkOTpZmZjk4\nWZqZ5eBkaWaWw/8BApvsoPEa6pIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11c203b90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "# Using Matplotlib. The confusion matrix that will result from this will be a heatmap. \n",
    "def plot_confusion_matrix(cm, title='Confusion matrix', cmap=plt.cm.Blues):\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('Actual label')\n",
    "    plt.xlabel('Predicted label')\n",
    "\n",
    "cm = metrics.confusion_matrix(y_test, predicted)\n",
    "np.set_printoptions(precision=2)\n",
    "print ('Confusion Matrix, without normalization')\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm)\n",
    "plt.show()\n",
    "\n",
    "cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "print('Normalized confusion matrix')\n",
    "print(cm_normalized)\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm_normalized, title='Normalized confusion matrix')\n",
    "\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 align='Left'>All Training Samples</h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 9min 16s, sys: 1.49 s, total: 9min 17s\n",
      "Wall time: 9min 19s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.89390000000000003"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regr2 = LogisticRegression(solver='sag')\n",
    "sample_size = len(train_dataset)\n",
    "X_train = train_dataset[:sample_size].reshape(sample_size, 784)\n",
    "y_train = train_labels[:sample_size]\n",
    "%time regr2.fit(X_train, y_train)\n",
    "regr2.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# http://scikit-learn.org/stable/auto_examples/linear_model/plot_iris_logistic.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "default_view": {},
   "name": "1_notmnist.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
